Efficient Evaluation of Classification and Recognition Systems

In this paper, a new framework for evaluating a variety of computer vision systems and components is introduced. This framework is particularly well suited for domains such as classification or recognition systems, where blind application of the i.i.d. assumption would reduce an evaulation’s accuracy, such as with classification or recognition systems. With a few exceptions, much of the previous work on vision system evaluation does not include confidence intervals, since they are difficult to calculate, and those that do are usually coupled with strict requirements. We show how a set of previously overlooked replicate statistics tools can be used to obtain tighter confidence intervals of evaluation estimates while simlutaneously reducing the amount of data and computation required to reach such sound evaluatory conclusions. In the included application of the new methodology, the well-known FERET face recognition system evaluation is extended to incorporate standard errors and confi-